Production Guidelines

Deploying Spark jobs to run reliably in production clusters (like Amazon EMR, Databricks, or Google Dataproc) requires moving beyond basic code syntax. You must master the architectural design patterns, monitoring systems, and resource configurations that separate amateur code from enterprise-grade pipelines.

The Medallion Architecture

The Medallion Architecture is a data design pattern that organizes data layers within a Lakehouse to incrementally clean and enrich datasets:

[Raw Sources]  Bronze (Raw Dump)  Silver (Cleansed/Enforced)  Gold (Aggregated Business BI)

1. Bronze Layer (Raw Data):
   • Goal: Securely capture and ingest data from source systems (APIs, databases, CDC logs) as fast as possible.
   • Format: Raw dumps, historically kept in Parquet or Delta formats. No schema checks or deduplications are applied.
2. Silver Layer (Cleansed & Enforced):
   • Goal: Clean, filter, match, and conform data.
   • Operations: Enforcing schemas, parsing JSON strings, applying joins, casting types, and removing duplicate records.
3. Gold Layer (Curated Business Insights):
   • Goal: Deliver highly aggregated, business-ready tables for dashboard reporting, SQL analysts, and machine learning models.
   • Operations: Computing aggregations (groupBy), window rankings, and analytical metrics.

Production Spark Monitoring & Logging

1. The Spark Web UI

The Spark UI is your primary tool to debug slow jobs:

• Active Stages Tab: If you see one task running for 2 hours while 99 tasks finished in 2 seconds, you have a Data Skew bottleneck.
• Executors Tab: Pay attention to GC Time. If Garbage Collection time is high (e.g. >10% of total run time), your executors are struggling with heap allocation. Switch to off-heap serialization or optimize caching.
• SQL Tab: Displays the graphical physical plan, allowing you to verify if Whole-Stage Code Gen is active (indicated by asterisks *) and check join strategies.

2. Standard Production Spark Configurations

Apply these resource-management settings inside your spark-submit shell scripts:

spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --driver-memory 4G \
  --executor-memory 8G \
  --num-executors 10 \
  --executor-cores 4 \
  --conf "spark.dynamicAllocation.enabled=true" \
  --conf "spark.eventLog.enabled=true" \
  --conf "spark.eventLog.dir=hdfs:///var/log/spark" \
  app.py

• --deploy-mode cluster: Driver process runs inside YARN on the cluster, avoiding network delays between your client machine and workers.
• spark.dynamicAllocation.enabled=true: Automatically spins up new executors during heavy operations and releases idle executors to save costs.
• spark.eventLog.enabled=true: Records execution history, enabling you to inspect job runs inside the Spark History Server after the cluster has shut down.

Congratulations: 10-Day Syllabus Complete!

You have completed the Spark Learning Plan! By progressing from low-level RDD cluster architectures to relational DataFrame DSL operations, parallelized file ingestions, advanced window analytics, DAG and Stage splits, caching optimization, and Delta Lake ACID transactions, you have built the foundational skills required to engineer robust, high-performance distributed data platforms.